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1 Introduction 



Suppose that iJ is a real separable Hilbert space. Given a Borel probability measure v on if, let 
V denote its characteristic functional. It is known that if v is infinitely divisible, then via) 7^ for 
all a G ii and there is a unique continuous function log u on H such that log i>(0) = and i>(a) = 
expjlog i>(a)}; see e.g. Linde j22l p. 20 and p. 58]. Let {Tt)t>o be a strongly continuous semigroup 
of linear operators on H with dual {T^)t>o and {nt)t>o a family of probability measures on 
H. The family {f^t)t>o is called a skew convolution semigroup (SC-semigroup) associated with 
{Tt)t>o if the following equation is satisfied: 

Hr+t = (Ttflr) * fJ-t, r,t>0, (1.1) 

where "*" denotes the convolution operation. It is easy to check that (jl.lj) holds if and only if 
we can define a Markov transition semigroup {Qi)t>o on H by 

Q'^f{x):= [ f{Ttx + y)Mdy), x G HJ G B{H), (1.2) 
Jh 

where B{H) denotes the totality of bounded Borel measurable functions on H. In this case, 
{Qt)t>o is called a generalized Mehler semigroup, which corresponds to a generalized Ornstein- 
Uhlenbeck process (OU-process) with state space H. This formulation of OU-processes was 
given by Bogachev et [S] as a generalization of the classical Mehler formula; see e.g. Malliavin 
[23l p. 17 and p. 25]. One motivation to study such OU-processes is that they constitute a large 
class of explicit examples of processes on infinite-dimensional spaces with rich mathematical 
structures. They arise in the study of Langevin type equations with generalized drift involving 
the generator of {Tt)t>o- We refer the reader to Bogachev et al jH], Puhrman and Rockner 
jl2j . and van Neerven ^21 for discussions from a theoretical viewpoint. See also Bogachev and 
Rockner j2j, Fuhrman and van Neerven |24| for some earlier related work. In the setting 
of cylindrical probability measures, Bogachev et al Lemma 2.6] proved that, if the function 
t ^ jj.t{a) is absolutely continuous on [0, 00) and differentiable at t = for all a £ H, then (ll.lj) 
is equivalent to 

/it(a) = expj - ^ A(r»fis|, t>0,aeH, (1.3) 

where A(a) = —{d/dt)jlt{a)\t=o is a negative-definite functional on H. A necessary and sufficient 
condition for a Gaussian SC-semigroup to be differentiable was given in van Neerven |22]. These 
results give characterizations for interesting special classes of SC-semigroups defined by p.l|) 
and have stimulated the present work. 

Skew convolution semigroups have also played an important role in the study of immigration 
structures associated with branching processes. Let E' be a Lusin topological space, i.e., a 
homeomorph of a Borel subset of a compact metric space, with Borel cr-algebra ,'^{E). We 
denote by B{E)^ the set of bounded non-negative Borel functions on E. Let M{E) be the 
totality of finite measures on (E,^{E)) endowed with the topology of weak convergence and 
{Qt)t>o the transition semigroup of a measure-valued branching process (superprocess) X with 
state space M(E). A family {Nt)t>o of probability measures on M{E) is called a SC-semigroup 
associated with {Qt)t>o if it satisfies 

Nr+t = (NrQt) * Nt, r,t>0. (1.4) 
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We use the same terminology for solutions of and those of (|1.4jl since p.lj) is actually a 
special form of H1.4I) when they are put in a slightly more general setting, say, when H and 
M{E) are replaced by a topological semigroup. This similarity between the two equations was 
first noticed by L.G. Gorostiza (1999, personal communication); see also Bojdecki and Gorostiza 
^ and Schmuland and Sun (22^. It is not hard to show that (|1.4j) holds if and only if 

:=Qt(z^,-)*iVt, t>0,zvGM(^) (1.5) 

defines a Markov semigroup {Q^)t>Q on M{E). A Markov process Y in M{E) is called an 
immigration process associated with X if it has transition semigroup {Qt^)t>o- The intuitive 
meaning of the immigration process is clear from (jl.Sj) . that is, Qt{i^,-) is the distribution of 
descendants of the people distributed as G M{E) at time zero and Nt is the distribution of 
descendants of the people immigrating to E during the time interval (0, t\. By Li jl7l Theorem 2] 
or |2H Theorem 3.2], the family {Nt)t>o satisfies (|1.4|) if and only if there is an infinitely divisible 
probability entrance law {Ks)s>o for {Qt)t>o such that 

log / e-^'^f^Ntidv) = f 
Jm{e) Jo 

where i^i f) = fdv; see also Li jl9l I21j for some generalizations of this result. Then there 
is a 1-1 correspondence between SC-semigroups and a set of infinitely divisible probability en- 
trance laws. Some representations of the infinitely divisible probability entrance laws and path 
regularity of the corresponding immigration processes were studied in Li TSj. The connection 
between immigration processes and generalized OU-processes was studied in Gorostiza and Li 
[3113] and Li j20^. In view of ((O)) . the function 

t^log / e-"^^^ Ntidv) (1.7) 

JM(E) 

is always absolutely continuous on [0,oo), and it is differentiable at t = for all continuous 
/ G B{E)^ if and nearly only if {Ks)s>o is closable by an infinitely divisible probability measure 
Kq on M{E). By the similarity of Hl.l|) and ()1.4|) . one might expect similar results for the 
solutions of (|1.H) . However, the Hilbert space situation is much more complicated as Schmuland 
and Sun j27j showed that the linear part of 1 1-^ log jUj(a) can be discontinuous. Therefore, we can 
only discuss characterizations for the solutions of (|l.lj) under reasonable regularity conditions 
on the linear part of t ^ log/ij(a). 

This work is also related to the catalytic branching superprocess introduced by Dawson 
and Fleischmann [21 El- Let us consider the special case where the underlying motion is an 
absorbing barrier Brownian motion (ABM) in a domain D. Let {Pt)t>o denote the transition 
semigroup of the ABM. Let r] G M{D) and let (/>(-, •) be a function on D x [0, oo) of a certain form 
to be specified. A catalytic branching superprocess in M{D) has transition semigroup {Qt)t>Q 
determined by 

/ e-^(^)Qt(^,di/) =exp{-/.(14/)}, fGBiD)+, (1.8) 
Jm{d) 

where {Vt)t>o is a semigroup of non-linear operators on B{D)^ defined by 

Vtf{x) = Ptf{x) - f ds j 0(y, Vsf{y))pt-s{x, yHdy), t>0,xeD, (1.9) 

Jo J D 



log 



M(E) 



-"^f'^Ksidi^) 



ds, t>0,f£B{Ey 



(1.6) 
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with pt{x, y) being the density of Pt{x, dy). This process describes the catalytic reaction of a large 
number of infinitesimal particles moving in D according to the transition law of the ABM and 
splitting according to the branching mechanism given by •). The measure r]{dx) represents 
the distribution of a catalyst in D which causes the splitting. More detailed descriptions of the 
model will been given in Section 3. 

In this paper, we give a representation for the general SC-semigroup {iJ-t)t>o defined by 
whose characteristic functional is not necessarily differentiable at t = 0. This result extends 
the interesting characterizations given in W and [^. The general representation is of interest 
since it includes some SC-semigroups arising in applications which are not included in (|1.3)) . We 
provide a rich class of SC-semigroups of this type in the case where H = L^(0, oo) and {Tt)t>o 
is the transition semigroup of the ABM. Indeed, the corresponding generalized OU-processes 
arise naturally as fluctuation limits of catalytic branching superprocesses with immigration. 
An important feature of these OU-processes is that they usually do not have right continuous 
realizations, which is similar to the situation of immigration processes studied in I19L OT] . 
Nevertheless, we show that some of these OU-processes are in fact quite regular if we regard them 
as processes with values of signed- measures. The study of generalized Mehler semigroups on 
Hilbert spaces and that of catalytic branching processes have evolved independently of each other 
with different motivations, techniques, and so on. The fluctuation limits establish a connection 
between the two subjects. 

The remainder of this paper is organized as follows. In Section 2 we give the characterization 
for general SC-semigroups. Fluctuation limits of immigration processes are studied in Section 3, 
which lead to generalized OU-processes with distribution values. Under stronger assumptions, it 
is proved in Section 4 that some of these OU-processes actually live in the Hilbert space L^{D) 
of functions. Regularity properties of the processes in the space of signed-measures are discussed 
in Section 5. 

2 Characterization of SC-semigroups 

In this section, we give a general representation of the SC-semigroups defined by (dHJ). It was 
proved in Schmuland and Sun ^\ that, if {^it)t>o is a solution of (jLlj) . then each fit is an 
infinitely divisible probability measure. Let 



By Linde I22J p. 75 and p. 84], the characteristic functional of fj,t on H is given by 

fit{a) = exp i^i{bt,a) - ^{Rta,a) + j K{x,a)Mt{dx)^ , a£H, (2.1) 

where bt G H, Rt is a symmetric, positive-definite, nuclear operator on H, and Mt is a cr-finite 
measure (Levy measure) on H° := H \ {0} satisfying 



Jh° 

Thus, fit is uniquely determined by the triple {bt, Rt, Mt) and is uniquely decomposed into the 
convolution of three infinitely divisible probabilities fit = fJ^t * fJ't * f^t with 



Kix,a) := -l-i{x,a)x[o,i]i\\x\\) 



x,a E H. 




(2.2) 





(2.3) 
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and 

/i^'(a) =exp|y" K{x,a)Mtidx)^ (2.4) 

for a £ H. We call /i^ the constant (or linear) part, fif the Gaussian part, and the jump part 
of ^t. By the uniqueness of the decomposition (|2.1() it is not hard to show that (jLlj) holds if 
and only if we have 

Rr+t = TtRr-T* + Rt, Mr+t = (TtM,)|H° + Mt, (2.5) 

and 

br+t = bt + Ttbr+ [ {x[o,i]mtx\\)-X[o,i]i\\x\\))TtxMridx) (2.6) 

for all r,t>0. 

Theorem 2.1 If {^t)t>Q is an SC-semigroup with decomposition i|2.I|) . then we can write 

{Rta,a) = [ {Usa,a)ds, t>0,aeH, (2.7) 
Jo 

where {Us)s>o is a family of nuclear operators on H satisfying Us+t = TfUsT^ for all s,t > and 




TrUsds <oo, t>0. 



The basic idea of the proof of this theorem is similar to that of ||17t Theorem 2], but the 
argument in the present case is more involved. We first prove two lemmas. 

Lemma 2.1 Under the conditions of Theorem 12.11 the function t {Rta,b) is absolutely 
continuous in t > for all a,b G H. 

Proof. If {fJ,t)t>o is an SC-semigroup, so is (^f)t>o by the first equation in (|2.5|) . Then we have 

/ \\xffi^r+tidx)= [ \\Ttxffi3{dx)+ [ \\x\\'^^lfidx), r,t>0. (2.8) 
Jh Jh Jh 

It follows that 

g(t) := / \\xffif{dx), t > (2.9) 
Jh 

is a non-decreasing function. Since (Tt)t>Q is strongly continuous, there are constants c > 1 and 
6 > such that ||Tj|| < ce^*. We claim that, for < ri < ti < • • • < < < /, 

n 

J2[g{tj)-g{r,)]<c'e'''g{a^), (2.10) 
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where an = Yl^=ii^j ~ ^i)- When n = 1, this follows from (|2.8j) . Now assume that (|2.1())) holds 
for n — 1. Applying H2.8() twice, 

f^iaitj) - 9{r,)] < [g{tn)-g{rn)]+c'e^'' j ||x|| V^., (^x) 

= [ ||T,„x||VL-.„(dx) + c2e2^' / ||x||VL_^(dx) 
Jh Jh 

Jh Jh 



Jh 

which gives (l2ln|l . Letting r ^ and t ^ in ((231) and using the fact that g is a non-decreasing 
function one sees that g{t) ^ as t ^ 0. By this and H2.1U|) . g is absolutely continuous in t > 0. 
From (|2.5|) we see that {Rta, a) is a non-decreasing function of t > for any a & H. For 
t > r > 0, ((Zni) yields 

{Rta, a) — {Rr a, a) = {Rt-rT*a,T*a) = / {x,T*a)'^ fif_j.{dx) 

Jh 

< \\af [ \\Trxfnf_,{dx) = \\af[g{t)-g{r)]. 
Jh 

Then {Rta, a) is absolutely continuous in t > 0. Polarization shows that {Rta, b) is absolutely 
continuous in t > for all a,b £ H . □ 

Lemma 2.2 Under the condition of Theorem \2.1l there is a family of nuclear operators {Us)s>o 
on H such that 1^2. 7\) holds. 

Proof. Let {e^ : n = 1, 2, . . .} be an orthonormal basis of H. By Lemma \2.1\ there are locally 
integrable functions Am,n on [0, cxd) such that 

{Rtem,en)= / Am,n{s)ds, t>0, m,n>l. (2.11) 
Jo 

From the symmetry of Rt we get 

[ Am,n(.s)ds= [ An,m{s)ds, (2.12) 

Jo Jo 
while the positivity of Rt gives 

„t oo 

{Rta,a) = ^ Ajn,n{s){a,ejn){a,en)ds > (2.13) 

•^0 rn.,n=l 

for a G spanjei, 62, . . .}. (The sum is actually finite!) In addition, since Rt is nuclear we have 
/ ( J^A„,n(s)jds = ^(i?te„,e„) =Tr(i?t) <cx). (2.14) 

^n=l ^ n=l 
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Let F be the Borel subset of [0, cxd) consisting of all s > such that Am,n{s) = An^m{s) for 
m, n > 1 and 

oo oo 

^m,n(s)(a, em)(a, e„) > and ^y4„,„(s)<oo (2.15) 

m,n=l n=l 

for a £ spanjei, 62, . . .} with rational coefficients. As observed in the proof of Lemma 12. H 
{Rta, a) is a non-decreasing function of t > 0. By ()2.12|) . 1)2. 13(1 and (|2.14l) . F has full Lebesgue 
measure. For any s G F, 

oo 

Usa= ^ Am,nis){a,em)en, (2.16) 

m,n=l 

defines a positive-definite, symmetric linear operator on spanjei, 62, • • •}• Taking b = xem + ye-n, 
with X, y rational, we get 

ITT i. i.\ r \ I ■Am.mySj A.m,n\s) \ f X \ 

^""''^'^ = [ A^,M An As) JUJ-°' 

SO that the 2x2 matrix above is non-negative definite. Therefore, its determinant is non-negative, 
that is, 

^m,n('S) ^ ylm,m('5)^n,n("5)- (^•1''') 

Combined with the Cauchy-Schwarz inequality this gives, 

oo ^ oo \2 

WUsaf' = ( Am.,n(.s)(Q, Cm) j 

n=l ^m=l ^ 
oo / oo ^ 2 

< ^A„,„(s)f ^^^,^(5)i/2|(a,e^)|j 

n=l ^ m=l ^ 

2 

2 



. oo ^ z 

< ( ^A„,„(s) J ||a 



for s G F and a G spanjei, 62, . . .}. This shows that Ug is a bounded operator and can be 
extended to the entire space H. In fact, Ug is a nuclear operator since 

00 00 
Ti{Us) = '^{Usen, e„) = A„,„(s) < 00. 

n=l n=\ 
By ()2.11|) and 1)2. 16() . for a G spanjei, 62, . . .} we have 

00 „t 
{Rta,a) = ^ {a,em){a,en){Rtem,en) = {Usa,a)ds, t>0. (2.18) 

m,n=l •^f' 

Since s 1-^ Ti(Us) is locally integrable, by dominated convergence we see that ()2.18|) holds for 
all a £ H. For s ^ F, we let Us be the zero operator. □ 



7 



Proof of Theorem 2.1. Let {Us)s>q be provided by Lemma 2.2. Note that (|2.7|) and the first 
equation of (|2.5|1 imply 

I {Us-{-ta,a)ds = / {UsT*a,T*a)ds, r,t>0,a€H. 
Jo Jo 

Since H is separable, by Fubini's theorem, there are subsets G and Gs of [0, oo) with full Lebesgue 
measure such that 

Us+t = TtUsT:, s£G,t£Gs. 

Choose a decreasing sequence s„ G G with s„ — > 0, and define 

Ut:=Tt^s„Us„T:_^^, t>sn. 

Under this modification, {Ut)t>o satisfies Ur+t = TtUrT^ for ah r, t > 0, while (|2.7|) remains 
unchanged. □ 

Theorem 2.2 If {fit)t>o is an SC-semigroup with decomposition i|2.1|) . then we can write 

[ K{x,a)Mt{dx) = [ ds [ K{x,a)Ls{dx), t>0,aeH, (2.19) 
Jh° Jo Jh° 

where Ls{dx) is a a-hnite kernel from {0,oo) to H° satisfying Lr-\-t = (TtLr)\H° for all r,t > 
and 



[ ds [ (1 A ||2;||^)L^((ix) < oo, t>0. 
Jo Jh 



Proof. If {nt)t>o is an SC-semigroup given by l\'2.1^ . then 1 1— > Mt is non-decreasing by the second 
equation in (|2.5() . Let c > 1 and 6 > be as in the proof of Lemma l2.ll and let 

h{t) := I (1 A \\xf)Mt{dx), t > 0. 
Jh° 

By ()2.5|) we have, for r, t > 0, 

h{r + t)-h{r)= [ {1 A \\Trxf)Mt{dx), 
Jh° 

which is bounded above by c^e'^^^h{t). As in the proof of Lemma 12.11 one sees that h{t) is 
absolutely continuous in t > 0. Since the family of finite measures i^t{dx) := (1 A ||x|p)Mt((ix) 
is non-decreasing and t i— > h{t) = vt{H°) is absolutely continuous, i^([0, t],i?) = t't(-B) defines 
a locally bounded Borel measure v{-,B) on [0,oo) for each B E ^(H°). A monotone class 
argument shows that i'{A, •) is a Borel measure on H° for each A £ =^([0, co)), so that •) is 
a bimeasure. By 10, p. 502], there is a probability kernel Js{dx) from [0, oo) to H° such that 

u{A,B)= [ Js{B)u{ds,H°) = [ Js{B)dh{s)= [ Js{B)h' {s)ds, 
J A J A J A 
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where h'{s) is the Radon-Nikodym derivative of dh{s) relative to Lebesgue measure. Defining 
the cr-finite kernel Ls{dx) := (1 A \\x\\'^)~^h'{s)Js{dx) we obtain (|2.19|) . By the second equation 
of (|2.5j) one can modify the definition of {Lt)t>o so that L^+t = (TtLr)\H° is satisfied for all 
r,t>0. □ 

We say the linear part {bt)t>o of (|2.1|) is absolutely continuous if there exists an if- valued 
path {cs)s>o such that {bt, a) = Jq{cs, a)ds for all t > and a ^ H. The following theorem gives a 
Hilbert space version of Li [111 Theorem 2] or 21 , Theorem 3.2] and extends the characterization 
of Bogachev et al ^ Lemma 2.6 and Proposition 4.3]. 

Theorem 2.3 Suppose that {fJ-t)t>o is a family of probability measures on H. If there is a 
family of inhnitely divisible probabilities {i's)s>o such that Vr+t = TfU^ for all r,t > and 



wo J 

then {fJ-t)t>o is an SC-semigroup. Conversely, every SC-semigroup {fJ-t)t>o with absolutely con- 
tinuous linear part has representation yj.'JUp . 

Proof. If {nt)t>o is given by (|2.2U|) . it is clearly an SC-semigroup. Conversely, let (/it)t>o be an 
SC-semigroup and let {Us)s>o and {Ls)s>o be provided by Theorems 12 . 1 1 and 12 . 21 Suppose that 
(6f,a) = /g (c<j, a)(is. By (|2.6|) . we can modify the definition of {cs)s>o so that 



Then we have the result by letting Vg be the infinitely divisible probability defined by the triple 

{Cs,Us,Ls). □ 

We may call the family {i's)s>o in Theorem 12.31 an entrance law for {Tt)t>o- (More precisely, 
it is an entrance law for the deterministic Markov process {TfX : t > 0}, as, for example, in 
Sharpe j28j.^ If there is a probability measure z^o on H such that z/^ = Tgi'o for all s > 0, we say 
that (z^s)s>o is closable. In this case, the corresponding SC-semigroup {fJ-t)t>o is given by 



which belongs to the class (|1.3|) . This explains the connection of our characterization with that 
of Bogachev et al 3. . 

Theorem 12.31 gives a characterization for all SC-semigroups under the assumption of absolute 
continuity on the linear part {bt)t>o- This assumption cannot be removed since {bt)t>o can be 
discontinuous as pointed out in Schmuland and Sun 27J. The following example shows that it 
can even be continuous but nowhere differentiable. 

Example 2.1 Consider H = L^([0,27r)) and let Tt be the shift operator by t > (mod 
2tt). For t>0 and / G L^([0,27r)) set bt = {I — Tt)f. Then {6bt)t>o is a constant SC-semigroup. 
Taking the inner product against / we obtain 





(2.20) 





(2.21) 




oo 



11/11 



2 



l/(0)P 



2^|/(n)|2cos(nt) 



n=l 
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where / is the Fourier transform of /. Now let / be the function whose Fourier coefficients are 
given by 



fin) 



2-^/"^ if \n\ = 2^ A: > 1, 
otherwise. 



Then we have 



(/,6t) = 2-2^^2-'= cos(2'=t), 



k=l 



which is (up to a constant) Weierstrass's nowhere differentiable continuous function. 

Let us consider another important special type of SC-semigroup given by 1)2.11) under the 
assumption: 



|x|| A \\xf)Mt{dx) < oo, t > 0. 



(2.22) 



H° 



Since 1)2.2(1 holds automatically, (|2.22|) is only a first norm-moment condition on the restriction 
of Mt io {x ^ H : \\x\\ > 1}. We say the SC-semigroup {nt)t>o is centered if 

/ {x,a)fit{dx) =0, t>0,aeH. 
Jh 

In this case. Theorem 12.31 implies that 



/ij(a)=exp| / - l-{Usa,a) + I Ki{x,a)Ls{dx) 
I Jo L ^ Jh° 



ds 



t>0,a€H, 



(2.23) 



where 



Ki{x,a) := e**'^''"^ - 1 - i{x,a), x,a£ H. 



Construction and regularity of OU-processes defined by (|2.23() are discussed systematically in 
Dawson and Li |S]. 

The characterizations ()2.2U() and (|2.23() are of interest since they include some SC-semigroups 
arising naturally in applications which are not included in ()1.3j) and ((2.211) . We shall see in the 
next two sections that a rich class of such SC-semigroups arise in the study of fluctuation limits 
of catalytic branching superprocesses with immigration. Two particular examples are given 
below. We consider the Hilbert space L2(0,oo). Let 



gtix) 



'2TTt 



exp{-x^/2t}, t>0,xe 



and 



Pt{x,y) = gt{x - y) - gt{x + y), t > 0, x, y e (0, oo). (2.24) 
Then the transition semigroup {Pt)t>Q of the ABM in (0, oo) is defined by Pq/ = / ^-nd 



Ptf{x) 



Ptix, y)f{y)dy, t> 0,x e {0, oo). 



(2.25) 
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Let 



kt{y) = 2-\d/dx)vt{x,y)U=o+ = ygt{y)/t, t>0,y£ (0,oo). (2.26) 
It is not hard to check that 

oo /"OO 

kt{y)dt = l and kr+t{y) = / pt{x,y)kr{x)dx (2.27) 
Jo 

for r,t > and y G (0, oo) . 

Example 2.2 Let c > and xq > 0. By Theorem 14.11 there is a centered Gaussian SC- 
semigroup {fJ-t)t>o on L^(0,cxd) given by 



/it(/) = exp| -c^ Psfixofds^ t >0,/eL2(0,oo). 



(2.28) 



This is a special form of (|2.20|) and (|2.23l) with {i's)s>o defined by 

i>,(/) = exp{-cP,/(xo)'}, s>0,f eL\0,oo). (2.29) 

Observe that / i-^ Ps/(xo)^ is a well-defined functional on L^(0, oo) only for s > 0. Thus, the 
SC-semigroup ((T^ is not included in (jOl) and 

Example 2.3 Suppose that (1 V \u\)m{du) is a finite measure on R° := R \ {0} and let 

ip{z)= (e™^ - 1 - mz) m(dn), z E M. 

By Theorem Ol 

/x;(/) =exp|^V((A:„/))ds|, t > 0, / G l2(0, oo) (2.30) 

defines a centered SC-semigroup {pi^)t>o on L^(0, oo). By (|2.27jl one may check that {pi^)t>Q is 
included in 1)2. 2U() and 1)2.23(1 . Unless m(M°) = 0, this SC-semigroup is not included in ()1.3|) and 

(Eini). 



3 Fluctuation limits of superprocesses 

In this section, we discuss small branching fluctuation limits of catalytic branching superpro- 
cesses with immigration, which lead to a class of OU-processes taking distribution values. Similar 
fluctuation limits for superprocesses with function-valued catalysts have been discussed in Goros- 
tiza ^3] ) Gorostiza and Li j and Li [201 • We shall only give an outline of the arguments 

and refer the reader to the earlier papers for details. As pointed out in [20], the small branching 
fluctuation limit is typically equivalent to the high density and the large scale fluctuation limits. 
For simplicity, we restrict to the case where the underlying motion is an ABM in D := (0, oo). 
We write D instead of (0, oo) for the underlying space in the sequel since (0, oo) and [0, oo) will 
appear frequently with quite different meanings. This notation also suggests that some of the 
results can be modifled to the case where D \s a, more general domain in R'^. 
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Let M{D) denote the space of finite Borel measures on D endowed with the topology of 
weak convergence. Let {Bt : t > 0} be an ABM in D with transition semigroup {Pt)t>o defined 
by (|2.25j) . Let •) be a function on D x [0, oo) given by 

/>oo 

4>{x,z) = c{x)z'^ + (e^'^" - 1 + zti)m(x,dn), z>0,x£D, (3.1) 
Jo 

where c E B{D)^ and u^m{x,du) is a bounded kernel from D to (0,co). For any t] G B{D)^, 
there is a superprocess in M[D) with transition semigroup {Qt)t>o determined by 

/ e-'^(^)Qt(/i,di/)=exp{-/i(I4/)}, fGB{D)+, (3.2) 
Jm{d) 

where {Vt)t>o is a semigroup of non-linear operators on B{D)^ defined by 

Vtf{x) = Ptf{x)- f ds [ (P{y,Vsf{y)Hy)Pt-s{x,dy), t>0,xeD, (3.3) 

Jo JD 

see for example Dawson The superprocess describes the catalytic reaction of a large num- 
ber of infinitesimal particles moving according to the transition law of the ABM and splitting 
according to the branching mechanism given by (/>(•, •). The value r}{x) represents the density 
at x G D of a catalyst which causes the splitting. However, there are some catalytic reactions 
in which the catalyst is concentrated on a very small set and in that case the coefficient r]{-) 
has to be replaced by an irregular one, as in Pagliaro and Taylor ,26,. These lead to the study 
of a catalyst given not by a regular density function but rather by a measure r] S M{D) with 
ri{dx) := "catalytic mass in the volume element dx". Then we reformulate 1)3. 3p as 

Vtf{x) = Ptfix) - f ds I cl){y, Vsf{y))pt-s{x, y)7i{dy), t>0,xGD, (3.4) 
Jo Jd 

where pt{x,y) is given by (|2.24|) . A Markov process in M{D) with transition semigroup {Qt)t>o 
defined by (|3.2j) and (|3.4|) is called a catalytic branching super ^5Mwith parameters (??,(/>)• 

Let {ls{y) : s > 0, y > 0} be a continuous version of the local time of {Bt : t > 0}. Then 
K(r,t) = r]{lt) — r]{lr) defines an additive functional of {Bt ■ t > 0}. In view of (|2.24|) it is easy 
to check that 

E^{K{Q,t)]= f ds f ps{x,y)7j{dy) <v{l) [ -^ds < 7j{l),/2t/^. 
Jo Jd Jo v^tts 

Thus, K{r, t) is admissible in the sense of [HJ p. 49] and the existence of the catalytic branching 
super ABM follows by p. 52]; see also jT^]. The study of superprocesses with irregular catalysts 
was initiated by Dawson and Fleischmann IH] and there has been a considerable development 
in the theory since then; see Dawson and Fleischmann [2j for a recent survey. 

Set Kt{dx) = kt{x)dx. By 1)2. 27() . (Kt)t>o forms an entrance law for the underlying semigroup 
(Pt)j>o, that is, KrPt = Kr+t ^OT all r,t > 0. Let 

St{K, f) = Ktif) - f ds f ^{y, VJ{y))kt-s{yHdy), t > OJ € B{D)+. (3.5) 
Jo Jd 
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As in Li |E] one may see that 

[ e-^(/)Qf(/x,(ii/)=exp|-/i(yj)- /'*5,(K,/)dr|, f e B{D)+ (3.6) 
Jm{d) I Jo ) 

defines the transition semigroup {Qt)t>o of a Markov process {Yj : t > 0} in M{D), which we 
shall call a catalytic branching immigration super ABM with parameters {ri,(j),n). By (|3.4|) and 
(|3.5|) it is not hard to check that 



= Ptf{x) and ^.St{K,ef) 

=0+ 



=0+ 



which, together with ()3.6() . imply that 



lyimUf^, dv) = KPtf) + r ^^r{f)dr. (3.7) 

M(D) JO 

By (tTTTl) and (EZZI) it follows that if Yq = A, then E{Yt{f)} = A(/) for alH > and / G B{D)+, 
where A denotes Lebesgue measure. 

Now we consider a small branching fluctuation limit of the catalytic branching immigration 
ABM. For any > 0, let (^e(x, z) = (j){x, 9z) and ^e(-D) = {n-O'^X : ^ G M{D)}. Suppose that 
{Yt : i > 0} is a catalytic branching immigration ABM with parameters (r/, (/>e, k) and Yq = A. 
As observed above, we have E{Yi{f)^ = A(/) for all t > and / G B{D)^ . On the other hand, 
(t)e{x, z) ^ as 6* ^ 0. By ((231), and ((2SI) we have y/(/) ^ A(/) in distribution as ^ 0. 
We define the fluctuation process {Zf : t > 0} by 

= - A], t>0. (3.8) 

As in Gorostiza and Li (llj we see that {Zf : t > 0} is a centered signed-measure- valued Markov 
process with transition semigroup {Rf)t>o determined by 

e-'^(/)i??(/i, di.) = exp j - KOVfif/e)) + /* r?(,/.(0y/(//e)))ds 
.re(D) I Jo 

where {Vf)t>o is defined by 

Vffix)+ f ds [ My,v!f{y))pt-s{x,y)v{dy)=Ptf{x). 

Jo J D 

Let ^{D) be the space of infinitely differentiable functions f on D such that 



max sup 

0<fe<n„6Z) 



Jk 



< oo, n = 0,1,2,.... (3.9) 



Then •9'{D) topologized by the norms {||| • : n = 0, 1, 2, . . .} is a nuclear space. Let ,5^'{D) 
denote the dual space of S^{D). As in Jl], the finite-dimensional distributions of {Zf : t > 0} 
converge as ^ — > to those of an oS^'(D)-valued Markov process {Zj* : t > 0} with transition 
semigroup {R^)t>o determined by 

/ e-^(^)i?°(M,di.)=exp|-^(PJ)+ /\(<A(P./))d4' (3.10) 
J.y"{D) I Jo J 

Therefore, an OU-process with transition semigroup {R^)t>o is an approximation of the fluctu- 
ations of an immigration process around the average. 
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Theorem 3.1 Let <p{-, ■) he a function on D x M. with the representation 

(p{x,z) = -c{x)z'^ + {e""" -l-izu)m{x,du), x£D,z£R, (3.11) 
Jr° 

where c G B{D)~^ and {\u\ A \u\'^)m{x, du) is a bounded kernel from D to M° := R \ {0}. Then 
there is a transition semigroup {Rt)t>o on ,5^'{D) given by 

f e"'^f'>Rt{f^,du)=exp{ifi{Ptf)+ f\{ip{Psf))ds], t>0,/G^(Z)). (3.12) 

J.y"{D) I Jo J 

Proof. The semigroup {Rt)t>o can be obtained as in Jl] by considering the difference of two 
Markov processes with transition semigroups of the form (|3.10|) . □ 

Heuristicahy, an OU-process with transition semigroup {Rt)t>o is the mixture of the fluc- 
tuations of two immigration processes around their means. The branching mechanism of the 
processes is determined by the function ip{-, •) given by ()3.11|) and the distribution of catalysts 
in D that cause the branching is given by ry. A more singular transition semigroup is given by 
the following 

Theorem 3.2 Let (p be a function on M given by 

ip{z) = -cz^+ {e'''^ -l-iuz)m{du), zeR, (3.13) 

where c > and {\u\ A \u\'^)m{du) is a finite measure on M°. Then there is a transition semigroup 
{R[)t>o on y'{D) given by 

[ e''''^f^R[{fi,du) = exp{ifi{Ptf)+ [\{Ksif))ds], t>OJey{D). (3.14) 
J.y"{D) I Jo J 

Proof. This transition semigroup is obtained from the one in the last theorem by replacing 
(p{x,z) and r]{dx) in (|3.12|) respectively by ip{nz) and 6i/2n{dx) and letting oo. □ 

Roughly speaking, an OU-process with transition semigroup {R'^)t>o represents the fluctua- 
tions of a process over D that branches very actively only near the absorbing boundary. 

4 OU-processes with function values 

In this section, we show that, under suitable conditions, the OU-processes constructed in the 
last section take function values from L^(D,X). 

Suppose that rj is a finite measure on D and (p{-,-) is given by ()3.11|) with u^m{x, du) 
being a bounded kernel from D to M° = M \ {0}. Let W{ds, dx) be a white noise on [0, oo) x 
D with covariance measure 2c{x)dsr){dx) and N{ds, du, dx) be a Poisson random measure on 
[0, oo) xW X D with intensity dsm{x, du)rj{dx). Suppose that W[ds, dx) and N{ds, du, dx) are 
defined on some complete probability space (i?,^, P) and are independent of each other. Set 
N(ds, du, dx) = N(ds, du, dx) — dsm{x, du)r](dx). Then we have 
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Theorem 4.1 For each t > 0, the function 



ft 

70 



Zf{uj,y):= / / pt-six,y)W{uj,ds,dx) + / / / upt^s{x,y)N{uj,ds,du,dx) (4.1) 
JO Jd Jo Jr° J d 

is well-defined in the L?'{Q x D,P x X) sense and {Z^ : t > 0} is a Markov process with state 
space L'^{D,X), initial value zero and transition semigroup {Rt)t>o given by 

[ e'^'''f^Rt{g,dh) = exp{i{g,Ptf)+ [\{^{Psf))ds], feL\D,X). (4.2) 
Jl^{d,\) I Jo J 

Moreover, Zf{uj,y) can be chosen as a function of (t,ijj,y) belonging to L'^{[0,T]x fix D, XxPxX) 
for each T > 0. 



Proof. By the inequahty 

2vr(t-s)yK""^\ (t-s)/"^ 2y^nit-sy 

we have 



j^pt-s{x,yfdy < ^^-j^ — / exp| - 77^^ }dy 



and 



I^eI^(^I^ I^Pt^s(.x,y)W{ds,dx)^ jdy 

dy ds pt-s{x,yfc{x)r]{dx) 
D Jo Jd 

< I =ds I c{x)r]{dx) 
Jo y'TT{t-s) Jd 

< 00 



e\{ / upt^s{x,y)N{ds,du,dx)] \dy 

D I V JO J]R° Jd J 

dy 1 ds ri{dx) / u'^pt-s{x,y)'^m{x,du) 
D Jo Jd Jr° 

< I — ds I r]{dx) I u^m{x,du) 







D 



< 00. 



Then the right hand side of 1)4. 1() is weU-defined in the Lp'{Q x D,P x X) sense. By the same 
reasoning, we see that it is also weh-defined in the L^([0, T] x f2xD,XxPxX) sense. For any 
/ G L'^{D, A), we have 



Eexpl^ij^ j^Pt^J{x)W{ds,dx)^ =expl^- ds j ^c{x)[Pt-s 



f{x)Mdx] 



and 



£?exp<i III uPt^sf{x)N{ds,du,dx) 
[ Jo Jr° Jd 

exp < ds c{x)ri{dx) / {ex.p{iuPt-sf{x)} — 1 — iuPt-sf{x))m{x,du)>. 
[ Jo Jd Jr" J 
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Thus {Z^ :t>0} has the asserted one-dimensional distributions. If 5 G L^{D,\), then Ptg G 
L^{D, A) for all t > 0. Clearly, the distribution Rt{g, •) of Ptg + has characteristic functional 
given by (|4.2j) and {Rt)t>o is a transition semigroup on L^{D,X). The Markov property of 
{Z^ : t > 0} follows by a similar calculation of the characteristic functionals of the finite- 
dimensional distributions. □ 

Suppose that (/?(•) is given by (l^.l.Sj) with v?'m{du) being a finite measure on W . Set "f{dx) = 
(1-e-^ )dx for x G D. Let {B{t) : t > 0} be a one-dimensional Brownian motion with increasing 
process 2ct and N{ds, du) be a Poisson random measure on [0, 00) x W with intensity dsm{du). 
Suppose that {B{t) : t > 0} and N{ds,du) are defined on some complete probability space 
(J7, P) and are independent of each other. Set N{ds, du) = N{ds, du) — dsm{du). Then we 
have 

Theorem 4.2 For each t > 0, the function 

Zf{uj,y):= f kt^s{y)B{LO,ds)+ [ [ uh-s{y)N{uo,ds,du) (4.3) 







is well-defined in the L'^ x D , P x^) sense and{Z^ : t > 0} can be regarded as a Markov process 
with state space ^'{W), initial value zero and transition semigroup (-Rj)t>o given by i|.lJ4|) . 
Moreover, Z^{u;,y) can be chosen as a function of {t,uj,y) belonging to L'^{[0,T]x {]xD, XxPxj) 
for each T > 0. 



Proof. For any t > 0, 



Then we have 



t poo 2 

ks{y)^ds< / JL^e-y'/'^ds 







2tts^ lny'^ 



I^El^(^l\t-s{y)B{ds)'j Y^dy) 



= 2c I -f{dy) I h^s{y)'ds 
< 00 



D 



and 

I^e!^(^I^ l^^ukt-s{y)N{ds,du)^ ^idy) 

= / 7(dy) / kt-s{y)'^ds / u^m{du) 

Jd Jo Jr° 
< 00. 

Thus, the right hand side of 1)4.31) is well-defined in the L^(/2 x D,P x 'y) sense. Clearly, it is 
also well-defined in the -^^^([0, T] x J? x D, A x P x 7) sense. For any / e L?{D, A), we have 

Eexpl^iJ^ {kt-s,f)B{ds)^ =exp|-^ c{kt-s, f)^ds^ 
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and 

ft 



£?exp<i / / u{kt-s, f)N{ds,du) 
[ Jo Jr° 



exp<^ ds {exp{iu{kt^s,f)}-'i--iu{h^s,f))m{du)}. 





Therefore {Z^ : t > 0} has the correct one-dimensional distributions. The asserted Markov 
property follows by a calculation of the characteristic functionals of the finite-dimensional dis- 
tributions. □ 

Theorem 4.3 If (1 V \u\)m{du) is a Enite measure on M°, then for each t >0 the function 

Z?{y):= f [ ukt-s{y)N{ds,du) (4.4) 



belongs to L^{D, A) a.s. and {Z^ : t > 0} is a Markov process with state space L^{D, A), initial 
value zero and transition semigroup (-Rj)t>o given by 

e'^'''f^R[{g,dh)=exp\i{g,Ptf)+ f\{{ks, f))ds] , feL\D,X), (4.5) 

L^{D,\) I Jo J 



where (p is given by 1^3. 13\) with c = 0. 

Proof. For any t > 0, we have 

/■* /■°° 1 _ 

/ ks{y)ds = / ^=e "du, 



which is bounded in y > and dominated by 



1 e--du = ie-?''/^* 



yV2t 



2/9/ A/TT VTT 



for y > V^t- Therefore, 



t 

ds I ukt-s'm{du) 







belongs to L'^{D, A) under our assumption. Since kt G L^{D, A) for every t > and a.s. 







ukt-sN{ds, du) 



is a finite sum, we have Z^ E L'^{D,X) a.s. If 5 € L'^{D,X), then Ptg G L'^{D,X) for ah t > 
and the distribution Rt{g,-) of Ptg + Z^ has characteristic functional given by ()4.5() . Clearly, 
{Rt)t>o is 3- transition semigroup on L2(L), A). The Markov property of {Z^ : t > 0} follows by 
a calculation of the characteristic functionals of the finite-dimensional distributions. □ 

As in Li ^H] one may see that the generalized OU-processes given by ()4.2|) and 1)4. 5() usually 
do not have right continuous sample paths, neither do they have the strong Markov property. 
We shall prove in the next section that they do have those properties if we regard them as 
processes in another suitably chosen state space. 
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5 OU-processes with signed-measure values 



In this section, we show that some of the generahzed OU-processes given by (|4.2|) and (|4.5|) 
behave very regularly in the space of signed-measures. Indeed, from the proof of Theorem 15.11 
we know that they are essentially special forms of the immigration processes studied in Li (ITIIT^ . 

Given a locally compact metric space E, we denote by M{E) the space of finite Borel 
measures on E. Let {fn}'^=i be a dense subset of the space of all bounded uniformly continuous 
functions on E. We define the metric r(-, •) on M{E) by 

oo 

r(/x,zv) = ^2""(lA|M/«)-K/n)l), fi,iyeM{E). (5.1) 

n=l 

Clearly, this metric is compatible with the topology of weak convergence in M{E). Let S{E) = 
{/i^ — /i^ : fi^ , fi'^ £ M{E)} be the space of finite signed-measures on E. Define a metric •) 
on S{E) by 

with /z^ — fi~ = n and — = u}. (5-2) 

Then fin — > /^o in S{E) if and only if there are decompositions fin = fJ't ~ f^n ^^'^ /"o = /^o^ — /^o 
such that /x+ —>■ ^jlq and /z,^ ^jLq in M{E). Below, we shall consider the metric space {S{E),p) 
for E = (0,oo) or [0,oo). 

Suppose that is a finite measure, is given by 1)3. 11() with c(x) = 0, \u\m{x,du) is a 

bounded kernel from D to M°, and that N{ds, du, dx) is a Poisson random measure on [0, oo) x 
]R° X D with intensity dsm{x,du)ri(dx). Let N(ds,du,dx) = N(ds,du,dx) — dsm{x,du)rj{dx) 
and 

Yt{f):= f I [ uPt-sf{x)N(.ds,du,dx), t>OJeBiD). (5.3) 
Jo Jr° Jd 

Theorem 5.1 The process {Yt ■ t >0} defined by i|5.ij|) is an a.s. right continuous S{D)-valued 
strong Markov process with transition semigroup {Rt)t>o defined by 

[ e"'^^^Rt{fi,du)=exp{ifi{Ptf)+ f\{^{Psf))ds], f e B{D). (5.4) 

JsiD) I Jo J 

Proof. We define the positive part {1"^"*" : t > 0} {Yt : t > 0} by 

rt poo p 

Yt^if)-= / uPt-sfix)N{ds,du,dx), t>0,fGBiD). 
Jo Jo Jd 

By the assumptions, 

E{Y+{f)} = ds r,{dx) / uPt-sf{x)m{x,du) 
Jo Jd Jo 



< t\\f\\ri{D) svcp j um{x,du) 

xeD Jo 

< oo. 



oo 
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Then {Yf^ : t > 0} is a well-defined M(Z))-vaIued process, which is clearly a special form of the 
immigration process considered in JH] without branching. By jl81 Theorem 4.1], {Y^^ '■ t > 0} 
is a.s. right continuous. Similarly, the negative part {Y^^ : t > 0} of {It ■ t > 0} defined by 

ft rO 

Yfif) : 



f [ [ uPt-sf{x)N{ds,du,dx), t>OJeB{D) 
Jo J-odJd 



is also an a.s. right continuous immigration process. Then one can easily see that {Yt : t > 
0} defined by (|5.3() is an a.s. right continuous S'(-D)-valued Markov process with transition 
semigroup {Rt)t>o- The strong Markov property holds since {Rt)t>o is clearly Feller. □ 

Suppose that (p is given by (|.S.13|) with c = 0, \u\m{du) is a finite measure on R°, and 
N{ds, du) is a Poisson random measure on [0, oo) x W with intensity dsm{du). Let N{ds, du) = 
N{ds,du) — dsm{du), and 

Yt{f):= f [ UKt-s{f)N{ds,du), t>0,feB{D). (5.5) 
Jo Jr° 

By an argument similar to that in the proof of Theorem 15.11 we get 

Theorem 5.2 The process {Yt : t >0} defined by i)5.5|) is an S{D)-valued Markov process with 
transition semigroup {R[)t>o defined by 



[ e"''^f^R[{fi,diy) = e^p{tfi{Ptf)+ [\{Ks{f))ds], f e B{D). 

JS{D) I Jo J 



(5.6) 



As in 18 one can see that the process (|5.5|) does not have any right continuous modification. 
Observe that h{x) := (1 — e~^) is an excessive function of {Pt)t>o and 

T f( \ = l Kxy^Pt{hf){x) for t > and x > 0, , . 

\ 2Kt{hf) = {d/dx)Pt{hf){0+) fort>Oandx = ^ ^ 

defines the transition semigroup {Tt)t>o of a Markov process on [0,oo). 

Theorem 5.3 Let {Yt : t > 0} be defined by ^t({0}) = 0, and Zt{dx) = (1 - e-'')Yt{dx) 

for X > 0. Tiien {Zt : t > 0} is an S{[Q,oo)) -valued Markov process with transition semigroup 
{St)t>o defined by 

[ e"'^f'^St{fi,du)=exp\ifx{Ttf)+ [\{His{hf))ds], f e B{[0,oo)). (5.8) 

J5([0,oo)) I Jo J 

Moreover, {Zt : t > 0} has a right continuous strong Markov reahzation. 

Proof. The first assertion holds by Theorem 15.21 Observe that 

/ e"'^f'>St{fi,du) = e^p{tfi{Ttf )+ f ^{2-^TJ{0))ds], / G i?([0, oo)), 

Js{[0,oo)) I Jo ) 

by ()5.7|) and 1)5. 8|) . Then the second assertion follows from jl8l Theorem 4.1] as in the proof of 
Theorem 15.11 □ 
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